Perturbed Iterate Analysis for Asynchronous Stochastic Optimization

We introduce and analyze stochastic optimization methods where the input to each gradient updateis perturbed by bounded noise. We show that this framework forms the basis of a unified approachto analyze asynchronous implementations of stochastic optimization algorithms. In this framework,asynchronous stochastic optimization algorithms can be thought of as serial methods operating on noisyinputs. Using our perturbed iterate framework, we provide new analyses of the Hogwild! algorithmand asynchronous stochastic coordinate descent, that are simpler than earlier analyses, remove manyassumptions of previous models, and in some cases yield improved upper bounds on the convergencerates. We proceed to apply our framework to develop and analyze KroMagnon: a novel, parallel,sparse stochastic variance-reduced gradient (SVRG) algorithm. We demonstrate experimentally on a16-core machine that the sparse and parallel version of SVRG is in some cases more than four orders ofmagnitude faster than the standard SVRG algorithm.